Method and device for displaying a two-dimensional image of a viewed object simultaneously with an image depicting the three-dimensional geometry of the viewed object

ABSTRACT

A method and device for displaying a two-dimensional image of a viewed object simultaneously with an image depicting a three-dimensional geometry of the viewed object using a video inspection device is disclosed. The video inspection device displays a two-dimensional image of the object surface of a viewed object, and determines the three-dimensional coordinates of a plurality of surface points. At least one rendered image of the three-dimensional geometry of the viewed object is displayed simultaneously with the two-dimensional image. As measurement cursors are placed and moved on the two-dimensional image, the rendered image of the three-dimensional geometry of the viewed object is automatically updated.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of, and claims priority to, U.S.application Ser. No. 14/660,464, filed Mar. 17, 2015, entitled “METHODAND DEVICE FOR DISPLAYING A TWO-DIMENSIONAL IMAGE OF A VIEWED OBJECTSIMULTANEOUSLY WITH AN IMAGE DEPICTING THE THREE-DIMENSIONAL GEOMETRY OFTHE VIEWED OBJECT,” which is a continuation-in-part, and claims priorityto, of U.S. application Ser. No. 14/108,976 (now U.S. Pat. No.9,875,574), filed Dec. 17, 2013, entitled “METHOD AND DEVICE FORAUTOMATICALLY IDENTIFYING THE DEEPEST POINT ON THE SURFACE OF ANANOMALY,” which is a continuation-in-part of, and claims priority to,U.S. patent application Ser. No. 13/040,678 (now U.S. Pat. No.9,013,469), filed Mar. 4, 2011, and entitled “METHOD AND DEVICE FORDISPLAYING A THREE-DIMENSIONAL VIEW OF THE SURFACE OF A VIEWED OBJECT,”which are hereby incorporated by reference in their entireties.

BACKGROUND OF THE INVENTION

The subject matter disclosed herein relates to a method and device fordisplaying a two-dimensional image of a viewed object simultaneouslywith an image depicting a three-dimensional geometry of the viewedobject using a video inspection device.

Video inspection devices, such as video endoscopes or borescopes, can beused to inspect a surface of an object to identify and analyze anomalies(e.g., pits or dents) on the object that may have resulted from, e.g.,damage, wear, corrosion, or improper installation. In many instances,the surface of the object is inaccessible and cannot be viewed withoutthe use of the video inspection device. For example, a video inspectiondevice can be used to inspect the surface of a blade of a turbine engineon an aircraft or power generation unit to identify any anomalies thatmay have formed on the surface to determine if any repair or furthermaintenance is required. In order to make that assessment, it is oftennecessary to obtain highly accurate-dimensional measurements of thesurface and the anomaly to verify that the anomaly does not exceed orfall outside an operational limit or required specification for thatobject.

A video inspection device can be used to obtain and display atwo-dimensional image of the surface of a viewed object showing theanomaly to determine the dimensions of an anomaly on the surface. Thistwo-dimensional image of the surface can be used to generatethree-dimensional data of the surface that provides thethree-dimensional coordinates (e.g., (x, y, z)) of a plurality of pointson the surface, including proximate to an anomaly. In some videoinspection devices, the user can operate the video inspection device ina measurement mode to enter a measurement screen in which the userplaces cursors on the two-dimensional image to determine geometricdimensions of the anomaly. In many instances, the contour of a viewedfeature is difficult to assess from the two-dimensional image, makinghighly accurate placement of the cursors proximate to the anomalydifficult as it is difficult for the user to visualize the measurementbeing performed in three-dimensional space. This process may not alwaysresult in the desired geometric dimension or measurement of the anomalybeing correctly determined and can be time consuming.

The discussion above is merely provided for general backgroundinformation and is not intended to be used as an aid in determining thescope of the claimed subject matter.

BRIEF DESCRIPTION OF THE INVENTION

A method and device for displaying a two-dimensional image of a viewedobject simultaneously with an image depicting a three-dimensionalgeometry of the viewed object using a video inspection device isdisclosed. The video inspection device displays a two-dimensional imageof the object surface of a viewed object, and determines thethree-dimensional coordinates of a plurality of surface points. At leastone rendered image of the three-dimensional geometry of the viewedobject is displayed simultaneously with the two-dimensional image. Asmeasurement cursors are placed and moved on the two-dimensional image,the rendered image of the three-dimensional geometry of the viewedobject is automatically updated.

An advantage that may be realized in the practice of some disclosedembodiments of the method and device for displaying a two-dimensionalimage of a viewed object simultaneously with an image depicting athree-dimensional geometry of the viewed object is that the accuracy ofthe measurement is improved since the user is provided with anadditional and better perspective of the anomaly, and the time toperform the measurement of an anomaly is reduced.

In one embodiment, a method for inspecting an object surface of a viewedobject is disclosed. The method includes the steps of displaying atwo-dimensional image of the object surface on a display, determiningthe three-dimensional coordinates of a plurality of points on the objectsurface using a central processor unit, determining a rendered image ofthe three-dimensional geometry of at least a portion of the objectsurface using the central processor unit, simultaneously displaying thetwo-dimensional image and the rendered image on the display, placing aplurality of measurement cursors on the two-dimensional image using apointing device and displaying the plurality of measurement cursors onthe two-dimensional image on the display, displaying on the renderedimage a plurality of measurement identifiers corresponding to themeasurement cursors on the two-dimensional image, and determining ameasurement dimension of the object surface based on the locations ofthe plurality of measurement cursors on the two-dimensional image usingthe central processor unit.

In another embodiment, the method includes the steps of placing aplurality of measurement cursors on the rendered image using a pointingdevice and displaying the plurality of measurement cursors on therendered image on the display displaying on the two-dimensional image aplurality of measurement identifiers corresponding to the measurementcursors on the rendered image, and determining a measurement dimensionof the object surface based on the locations of the plurality ofmeasurement cursors on the rendered image using the central processorunit.

In yet another embodiment, the method includes the steps of displaying atwo-dimensional stereo image of the object surface on a display,determining the three-dimensional coordinates of a plurality of pointson the object surface using a central processor unit using stereotechniques, determining a rendered image of the three-dimensionalgeometry of at least a portion of the object surface using the centralprocessor unit, and simultaneously displaying the two-dimensional stereoimage and the rendered image on the display.

In still another embodiment, a device for inspecting an object surfaceof a viewed object is disclosed. The device includes an elongated probecomprising an insertion tube, an imager located at a distal end of theinsertion tube for obtaining a two-dimensional stereo image of theobject surface, a central processor unit for determining thethree-dimensional coordinates of a plurality of points on the objectsurface, and determining a rendered image of the three-dimensionalgeometry of at least a portion of the object surface, and a display forsimultaneously displaying the two-dimensional stereo image and therendered image.

This brief description of the invention is intended only to provide abrief overview of subject matter disclosed herein according to one ormore illustrative embodiments, and does not serve as a guide tointerpreting the claims or to define or limit the scope of theinvention, which is defined only by the appended claims. This briefdescription is provided to introduce an illustrative selection ofconcepts in a simplified form that are further described below in thedetailed description. This brief description is not intended to identifykey features or essential features of the claimed subject matter, nor isit intended to be used as an aid in determining the scope of the claimedsubject matter. The claimed subject matter is not limited toimplementations that solve any or all disadvantages noted in thebackground.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the features of the invention can beunderstood, a detailed description of the invention may be had byreference to certain embodiments, some of which are illustrated in theaccompanying drawings. It is to be noted, however, that the drawingsillustrate only certain embodiments of this invention and are thereforenot to be considered limiting of its scope, for the scope of theinvention encompasses other equally effective embodiments. The drawingsare not necessarily to scale, emphasis generally being placed uponillustrating the features of certain embodiments of the invention. Inthe drawings, like numerals are used to indicate like parts throughoutthe various views. Thus, for further understanding of the invention,reference can be made to the following detailed description, read inconnection with the drawings in which:

FIG. 1 is a block diagram of an exemplary video inspection device;

FIG. 2 is an exemplary image obtained by the video inspection device ofthe object surface of a viewed object having an anomaly in an exemplaryembodiment;

FIG. 3 is a flow diagram of an exemplary method for automaticallyidentifying the deepest point on the surface of an anomaly on a viewedobject shown in the image of FIG. 2 in an exemplary embodiment;

FIG. 4 illustrates an exemplary reference surface determined by thevideo inspection device;

FIG. 5 illustrates an exemplary region of interest determined by thevideo inspection device;

FIG. 6 illustrates another exemplary region of interest determined bythe video inspection device;

FIG. 7 is a graphical representation of an exemplary profile of theobject surface of the viewed object shown in the image of FIG. 1 in anexemplary embodiment;

FIG. 8 is another image obtained by the video inspection device of thesurface of a viewed object having an anomaly in an exemplary embodiment;

FIG. 9 is a flow diagram of a method for displaying three-dimensionaldata for inspection of the surface of the viewed object shown in theimage of FIG. 8 in an exemplary embodiment;

FIG. 10 is a display of a subset of a plurality of surface points in apoint cloud view;

FIG. 11 is a flow diagram of an exemplary method for displaying atwo-dimensional image of viewed object simultaneously with an imagedepicting the three-dimensional geometry of the viewed object in anotherexemplary embodiment;

FIG. 12 is a display of a two-dimensional image and a stereo image ofthe viewed object;

FIG. 13 is a display of a two-dimensional image of the viewed objectwith measurement cursors and a rendered image of the three-dimensionalgeometry of the viewed object in the form of a depth profile image withmeasurement identifiers; and

FIG. 14 is a display of a two-dimensional image of the viewed objectwith measurement cursors and a rendered image of the three-dimensionalgeometry of the viewed object in the form of a point cloud view withmeasurement identifiers.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is a block diagram of an exemplary video inspection device 100.It will be understood that the video inspection device 100 shown in FIG.1 is exemplary and that the scope of the invention is not limited to anyparticular video inspection device 100 or any particular configurationof components within a video inspection device 100.

Video inspection device 100 can include an elongated probe 102comprising an insertion tube 110 and a head assembly 120 disposed at thedistal end of the insertion tube 110. Insertion tube 110 can be aflexible, tubular section through which all interconnects between thehead assembly 120 and probe electronics 140 are passed. Head assembly120 can include probe optics 122 for guiding and focusing light from theviewed object 202 onto an imager 124. The probe optics 122 can comprise,e.g., a lens singlet or a lens having multiple components. The imager124 can be a solid state CCD or CMOS image sensor for obtaining an imageof the viewed object 202.

A detachable tip or adaptor 130 can be placed on the distal end of thehead assembly 120. The detachable tip 130 can include tip viewing optics132 (e.g., lenses, windows, or apertures) that work in conjunction withthe probe optics 122 to guide and focus light from the viewed object 202onto an imager 124. The detachable tip 130 can also include illuminationLEDs (not shown) if the source of light for the video inspection device100 emanates from the tip 130 or a light passing element (not shown) forpassing light from the probe 102 to the viewed object 202. The tip 130can also provide the ability for side viewing by including a waveguide(e.g., a prism) to turn the camera view and light output to the side.The tip 130 may also provide stereoscopic optics or structured-lightprojecting elements for use in determining three-dimensional data of theviewed surface. The elements that can be included in the tip 130 canalso be included in the probe 102 itself.

The imager 124 can include a plurality of pixels formed in a pluralityof rows and columns and can generate image signals in the form of analogvoltages representative of light incident on each pixel of the imager124. The image signals can be propagated through imager hybrid 126,which provides electronics for signal buffering and conditioning, to animager harness 112, which provides wires for control and video signalsbetween the imager hybrid 126 and the imager interface electronics 142.The imager interface electronics 142 can include power supplies, atiming generator for generating imager clock signals, an analog frontend for digitizing the imager video output signal, and a digital signalprocessor for processing the digitized imager video data into a moreuseful video format.

The imager interface electronics 142 are part of the probe electronics140, which provide a collection of functions for operating the videoinspection device 10. The probe electronics 140 can also include acalibration memory 144, which stores the calibration data for the probe102 and/or tip 130. A microcontroller 146 can also be included in theprobe electronics 140 for communicating with the imager interfaceelectronics 142 to determine and set gain and exposure settings, storingand reading calibration data from the calibration memory 144,controlling the light delivered to the viewed object 202, andcommunicating with a central processor unit (CPU) 150 of the videoinspection device 100.

In addition to communicating with the microcontroller 146, the imagerinterface electronics 142 can also communicate with one or more videoprocessors 160. The video processor 160 can receive a video signal fromthe imager interface electronics 142 and output signals to variousmonitors 170, 172, including an integral display 170 or an externalmonitor 172. The integral display 170 can be an LCD screen built intothe video inspection device 100 for displaying various images or data(e.g., the image of the viewed object 202, menus, cursors, measurementresults) to an inspector. The external monitor 172 can be a videomonitor or computer-type monitor connected to the video inspectiondevice 100 for displaying various images or data.

The video processor 160 can provide/receive commands, statusinformation, streaming video, still video images, and graphical overlaysto/from the CPU 150 and may be comprised of FPGAs, DSPs, or otherprocessing elements which provide functions such as image capture, imageenhancement, graphical overlay merging, distortion correction, frameaveraging, scaling, digital zooming, overlaying, merging, flipping,motion detection, and video format conversion and compression.

The CPU 150 can be used to manage the user interface by receiving inputvia a joystick 180, buttons 182, keypad 184, and/or microphone 186, inaddition to providing a host of other functions, including image, video,and audio storage and recall functions, system control, and measurementprocessing. The joystick 180 can be manipulated by the user to performsuch operations as menu selection, cursor movement, slider adjustment,and articulation control of the probe 102, and may include a push-buttonfunction. The buttons 182 and/or keypad 184 also can be used for menuselection and providing user commands to the CPU 150 (e.g., freezing orsaving a still image). The microphone 186 can be used by the inspectorto provide voice instructions to freeze or save a still image.

The video processor 160 can also communicate with video memory 162,which is used by the video processor 160 for frame buffering andtemporary holding of data during processing. The CPU 150 can alsocommunicate with CPU program memory 152 for storage of programs executedby the CPU 150. In addition, the CPU 150 can be in communication withvolatile memory 154 (e.g., RAM), and non-volatile memory 156 (e.g.,flash memory device, a hard drive, a DVD, or an EPROM memory device).The non-volatile memory 156 is the primary storage for streaming videoand still images.

The CPU 150 can also be in communication with a computer I/O interface158, which provides various interfaces to peripheral devices andnetworks, such as USB, Firewire, Ethernet, audio I/O, and wirelesstransceivers. This computer I/O interface 158 can be used to save,recall, transmit, and/or receive still images, streaming video, oraudio. For example, a USB “thumb drive” or CompactFlash memory card canbe plugged into computer I/O interface 158. In addition, the videoinspection device 100 can be configured to send frames of image data orstreaming video data to an external computer or server. The videoinspection device 100 can incorporate a TCP/IP communication protocolsuite and can be incorporated in a wide area network including aplurality of local and remote computers, each of the computers alsoincorporating a TCP/IP communication protocol suite. With incorporationof TCP/IP protocol suite, the video inspection device 100 incorporatesseveral transport layer protocols including TCP and UDP and severaldifferent layer protocols including HTTP and FTP.

It will be understood that, while certain components have been shown asa single component (e.g., CPU 150) in FIG. 1, multiple separatecomponents can be used to perform the functions of the CPU 150.

FIG. 2 is an exemplary image 200 obtained by the video inspection device100 of the object surface 210 of a viewed object 202 having an anomaly204 in an exemplary embodiment of the invention. In this example, theanomaly 204 is shown as a dent, where material has been removed from theobject surface 210 of the viewed object 202 in the anomaly 204 by damageor wear. It will be understood that the anomaly 204 shown in thisexemplary embodiment is just an example and that the inventive methodapplies to other types of irregularities (e.g., cracks, corrosionpitting, coating loss, surface deposits, etc.). Once the image 200 isobtained, and the anomaly 204 is identified, the image 200 can be usedto determine the dimensions of the anomaly 204 (e.g., height or depth,length, width, area, volume, point to line, profile slice, etc.). In oneembodiment, the image 200 used can be a two-dimensional image 200 of theobject surface 210 of the viewed object 202, including the anomaly 204.

FIG. 3 is a flow diagram of an exemplary method 300 for automaticallyidentifying the deepest point on the object surface 210 of an anomaly204 on a viewed object 202 shown in the image 200 of FIG. 2 in anexemplary embodiment of the invention. It will be understood that thesteps described in the flow diagram of FIG. 3 can be performed in adifferent order than shown in the flow diagram and that not all of thesteps are required for certain embodiments.

At step 310 of the exemplary method 300 (FIG. 3) and as shown in FIG. 2,the user can use the video inspection device 100 (e.g., the imager 124)to obtain at least one image 200 of the object surface 210 of a viewedobject 202 having an anomaly 204 and display it on a video monitor(e.g., an integral display 170 or external monitor 172). In oneembodiment, the image 200 can be displayed in a measurement mode of thevideo inspection device.

At step 320 of the exemplary method 300 (FIG. 3), the video inspectiondevice 100 (e.g., the CPU 150) can determine the three-dimensionalcoordinates (e.g., (x, y, z)) of a plurality of surface points on theobject surface 210 of the viewed object 202, including surface points ofthe anomaly 204. In one embodiment, the video inspection device cangenerate three-dimensional data from the image 200 in order to determinethe three-dimensional coordinates. Several different existing techniquescan be used to provide the three-dimensional coordinates of the surfacepoints in the image 200 (FIG. 2) of the object surface 210 (e.g.,stereo, scanning systems, stereo triangulation, structured light methodssuch as phase shift analysis, phase shift moiré, laser dot projection,etc.).

Most such techniques comprise the use of calibration data, which, amongother things, includes optical characteristic data that is used toreduce errors in the three-dimensional coordinates that would otherwisebe induced by optical distortions. With some techniques, thethree-dimensional coordinates may be determined using one or more imagescaptured in close time proximity that may include projected patterns andthe like. It is to be understood that references to three-dimensionalcoordinates determined using image 200 may also comprisethree-dimensional coordinates determined using one or a plurality ofimages 200 of the object surface 210 captured in close time proximity,and that the image 200 displayed to the user during the describedoperations may or may not actually be used in the determination of thethree-dimensional coordinates.

At step 330 of the exemplary method 300 (FIG. 3), and as shown in FIG.4, the video inspection device 100 (e.g., the CPU 150) can determine areference surface 250. In some embodiments, the reference surface 250can be flat, while in other embodiments the reference surface 250 can becurved. Similarly, in one embodiment, the reference surface 250 can bein the form of a plane, while in other embodiments, the referencesurface 250 can be in the form of a different shape (e.g., cylinder,sphere, etc.). For example, a user can use the joystick 180 (or otherpointing device (e.g., mouse, touch screen)) of the video inspectiondevice 100 to select one or more reference surface points on the objectsurface 210 of the viewed object 202 proximate to the anomaly 204 todetermine a reference surface.

In one embodiment and as shown in FIG. 4, a total of three referencesurface points 221, 222, 223 are selected on the object surface 210 ofthe viewed object 202 proximate to the anomaly 204 to conduct a depthmeasurement of the anomaly 204, with the three reference surface points221, 222, 223 selected on the object surface 210 proximate to theanomaly 204. In one embodiment, the plurality of reference surfacepoints 221, 222, 223 on the object surface 210 of the viewed object 202can be selected by placing reference surface cursors 231, 232, 233 (orother pointing devices) on pixels 241, 242, 243 of the image 200corresponding to the plurality of reference surface points 221, 222, 223on the object surface 210. In the exemplary depth measurement, the videoinspection device 100 (e.g., the CPU 150) can determine thethree-dimensional coordinates of each of the plurality of referencesurface points 221, 222, 223.

The three-dimensional coordinates of three or more surface pointsproximate to one or more of the three reference surface points 221, 222,223 selected on the object surface 210 proximate to the anomaly 204 canbe used to determine a reference surface 250 (e.g., a plane). In oneembodiment, the video inspection device 100 (e.g., the CPU 150) canperform a curve fitting of the three-dimensional coordinates of thethree reference surface points 221, 222, 223 to determine an equationfor the reference surface 250 (e.g., for a plane) having the followingform:

k _(0RS) +k _(1RS1) ·x _(iRS) +k _(2RS) ·y _(iRS1) =z _(iRS)  (1)

where (x_(iRS), y_(iRS), z_(iRS)) are coordinates of anythree-dimensional point on the defined reference surface 250 andk_(0RS), k_(1RS), and k_(2RS) are coefficients obtained by a curvefitting of the three-dimensional coordinates.

It should be noted that a plurality of reference surface points (i.e.,at least as many points as the number of k coefficients) are used toperform the curve fitting. The curve fitting finds the k coefficientsthat give the best fit to the points used (e.g., least squaresapproach). The k coefficients then define the plane or other referencesurface 250 that approximates the three-dimensional points used.However, if more points are used in the curve fitting than the number ofk coefficients, when you insert the x and y coordinates of the pointsused into the plane equation (1), the z results will generally notexactly match the z coordinates of the points due to noise and anydeviation from a plane that may actually exist. Thus, the x_(iRS1) andy_(iRS1) can be any arbitrary values, and the resulting z_(iRS) tellsyou the z of the defined plane at x_(iRS), y_(iRS). Accordingly,coordinates shown in these equations can be for arbitrary points exactlyon the defined surface, not necessarily the points used in the fittingto determine the k coefficients.

In other embodiments, there are only one or two reference surface pointsselected, prohibiting the use of curve fitting based only on thethree-dimensional coordinates of those reference surface points sincethree points are needed to determine k_(0RS), k_(1RS), and k_(2RS). Inthat case, the video inspection device 100 (e.g., the CPU 150) canidentify a plurality of pixels proximate to each of the pixels of theimage corresponding to a plurality of points on the object surface 210proximate to the reference surface point(s), and determine thethree-dimensional coordinates of the proximate point(s), enabling curvefitting to determine a reference surface 250.

While the exemplary reference surface 250 has been described as beingdetermined based on reference surface points 221, 222, 223 selected byreference surface cursors 231, 232, 233, in other embodiments, thereference surface 250 can be formed by using a pointing device to placea reference surface shape 260 (e.g., circle, square, rectangle,triangle, etc.) proximate to anomaly 204 and using the reference surfacepoints 261, 262, 263, 264 of the shape 260 to determine the referencesurface 250. It will be understood that the reference surface points261, 262, 263, 264 of the shape 260 can be points selected by thepointing device or be other points on or proximate to the perimeter ofthe shape that can be sized to enclose the anomaly 204.

At step 340 of the exemplary method 300 (FIG. 3), and as shown in FIG.5, the video inspection device 100 (e.g., the CPU 150) determines aregion of interest 270 proximate to the anomaly 204 based on thereference surface points of the reference surface 250. The region ofinterest 270 includes a plurality of surface points of the anomaly 204.In one embodiment, a region of interest 270 is formed by forming aregion of interest shape 271 (e.g., a circle) based on two or more ofthe reference surface points 221, 222, 223. In another embodiment, theregion of interest 270 can be determined by forming a cylinderperpendicular to the reference surface 260 and passing it through orproximate to two or more of the reference surface points 221, 222, 223.Referring again to FIG. 4, a region of interest could be formed withinthe reference surface shape 260 and reference surface points 261, 262,263, 264.

Although the exemplary region of interest shape 271 in FIG. 5 is formedby passing through the reference surface points 221, 222, 223, inanother embodiment, a smaller diameter reference surface shape can beformed by passing only proximate to the reference surface points. Forexample, as shown in FIG. 6, a region of interest 280 is formed bypassing a region of interest shape 281 (e.g., a circle) proximate to twoof the reference surface points 221, 222, where the diameter of thecircle 281 is smaller than the distance between the two referencesurface points 221, 222. It will be understood that region of interestshapes 271, 281 and the regions of interest 270, 280 may or may not bedisplayed on the image 200.

After the region of interest 270, 280 is determined, at step 350 of theexemplary method 300 (FIG. 3), the video inspection device 100 (e.g.,the CPU 150) determines the distance (i.e., depth) from each of theplurality of surface points in the region of interest to the referencesurface 250. In one embodiment, the video inspection device 100 (e.g.,the CPU 150) determines the distance of a line extending between thereference surface 250 and each of the plurality of surface points in theregion of interest 270, 280, wherein the line perpendicularly intersectsthe reference surface 250.

At step 360 of the exemplary method 300 (FIG. 3), the video inspectiondevice determines the location of the deepest surface point 224 in theregion of interest 270, 280 by determining the surface point that isfurthest from the reference surface 250 (e.g., selecting the surfacepoint with the longest line extending to the reference surface 250). Itwill be understood that, as used herein, the “deepest point” or “deepestsurface point” can be a furthest point that is recessed relative to thereference surface 250 or a furthest point (i.e., highest point) that isprotruding from the references surface 250. The video inspection device100 can identify the deepest surface point 224 in the region of interest270, 280 on the image by displaying, e.g., a cursor 234 (FIG. 5) orother graphic identifier 282 (FIG. 6) on the deepest surface point 224.In addition and as shown in FIGS. 5 and 6, the video inspection device100 can display the depth 290 (in inches or millimeters) of the deepestsurface point 224 in the region of interest 270, 280 on the image 200(i.e., the length of the perpendicular line extending from the deepestsurface point 224 to the reference surface 250. By automaticallydisplaying the cursor 234 or other graphic identifier 282 (FIG. 6) atthe deepest surface point 224 in the region of interest 270, 280, thevideo inspection device 100 reduces the time required to perform thedepth measurement and improves the accuracy of the depth measurementsince the user does not need to manually identify the deepest surfacepoint 224 in the anomaly 204.

Once the cursor 234 has been displayed at the deepest surface point 224in the region of interest 270, 280, the user can select that point totake and save a depth measurement. The user can also move the cursor 234within the region of interest 270, 280 to determine the depth of othersurface points in the region of interest 270, 280. In one embodiment,the video inspection device 100 (e.g., CPU 150) can monitor the movementof the cursor 234 and detect when the cursor 234 has stopped moving.When the cursor 234 stops moving for a predetermined amount of time(e.g., 1 second), the video inspection device 100 (e.g., the CPU 150)can determine the deepest surface point proximate to the cursor 234(e.g., a predetermined circle centered around the cursor 234) andautomatically move the cursor 234 to that position.

FIG. 7 is a graphical representation of an exemplary profile 370 of theobject surface 210 of the viewed object 202 shown in the image 200 ofFIG. 1. In this exemplary profile 370, the reference surface 250 isshown extending between two reference surface points 221, 222 and theirrespective reference surface cursors 231, 232. The location and depth290 of the deepest surface point 224 in the region of interest is alsoshown in the graphical representation. In another embodiment, a pointcloud view can also be used to show the deepest surface point 224.

FIG. 8 is another image 500 obtained by the video inspection device 100of the object surface 510 of a viewed object 502 having an anomaly 504in an exemplary embodiment of the invention. Once again, in thisexample, the anomaly 504 is shown as a dent, where material has beenremoved from the object surface 510 of the viewed object 502 in theanomaly 504 by damage or wear. It will be understood that the anomaly504 shown in this exemplary embodiment is just an example and that theinventive method applies to other types of irregularities (e.g., cracks,corrosion pitting, coating loss, surface deposits, etc.). Once the image500 is obtained, and the anomaly 504 is identified, the image 500 can beused to determine the dimensions of the anomaly 504 (e.g., height ordepth, length, width, area, volume, point to line, profile slice, etc.).In one embodiment, the image 500 used can be a two-dimensional image 500of the object surface 510 of the viewed object 502, including theanomaly 504.

FIG. 9 is a flow diagram of a method 600 for displayingthree-dimensional data for inspection of the object surface 510 of theviewed object 502 shown in the image 500 of FIG. 8 in an exemplaryembodiment of the invention. It will be understood that the stepsdescribed in the flow diagram of FIG. 9 can be performed in a differentorder than shown in the flow diagram and that not all of the steps arerequired for certain embodiments.

At step 610, and as shown in FIG. 8, the operator can use the videoinspection device 100 to obtain an image 500 of the object surface 510of a viewed object 502 having an anomaly 504 and display it on a videomonitor (e.g., an integral display 170 or external monitor 172). In oneembodiment, the image 500 can be displayed in a measurement mode of thevideo inspection device.

At step 620, the CPU 150 of the video inspection device 100 candetermine the three-dimensional coordinates (x_(iS1), y_(iS1), z_(iS1))in a first coordinate system of a plurality of surface points on theobject surface 510 of the viewed object 502, including the anomaly 504.In one embodiment, the video inspection device can generatethree-dimensional data from the image 500 in order to determine thethree-dimensional coordinates. As discussed above, several differentexisting techniques can be used to provide the three-dimensionalcoordinates of the points on the image 500 of the object surface 510(e.g., stereo, scanning systems, structured light methods such as phaseshifting, phase shift moiré, laser dot projection, etc.).

At step 630, and as shown in FIG. 8, an operator can use the joystick180 (or other pointing device (e.g., mouse, touch screen)) of the videoinspection device 100 to select a plurality of measurement points on theobject surface 510 of the viewed object 502 proximate the anomaly 504 toconduct a particular type of measurement. The number of measurementpoints selected is dependent upon the type measurement to be conducted.Certain measurements can require selection of two measurement points(e.g., length, profile), while other measurements can require selectionof three or more measurement points (e.g., point-to-line, area,multi-segment). In one embodiment and as shown in FIG. 8, a total offour measurement points 521, 522, 523, 524 are selected on the objectsurface 510 of the viewed object 502 proximate the anomaly 504 toconduct a depth measurement of the anomaly 504, with three of themeasurement points 521, 522, 523 selected on the object surface 510proximate the anomaly 504, and the fourth measurement point 524 selectedto be at the deepest point of the anomaly 504. In one embodiment, theplurality of measurement points 521, 522, 523, 524 on the object surface510 of the viewed object 502 can be selected by placing cursors 531,532, 533, 534 (or other pointing devices) on pixels 541, 542, 543, 544of the image 500 corresponding to the plurality of measurement points521, 522, 523, 524 on the object surface 510. In the exemplary depthmeasurement, the video inspection device 100 can determine thethree-dimensional coordinates in the first coordinate system of each ofthe plurality of measurement points 521, 522, 523, 524. It will beunderstood that the inventive method is not limited to depthmeasurements or measurements involving four selected measurement points,but instead applies to various types of measurements involving differentnumbers of points, including those discussed above.

At step 640, and as shown in FIG. 8, the CPU 150 of the video inspectiondevice 100 can determine a reference surface 550. In the exemplary depthmeasurement of the anomaly 504 shown in FIG. 8, the three-dimensionalcoordinates of three or more surface points proximate one or more of thethree measurement points 521, 522, 523 selected on the object surface510 proximate the anomaly 504 can be used to determine a referencesurface 550 (e.g., a plane). In one embodiment, the video inspectiondevice 100 can perform a curve fitting of the three-dimensionalcoordinates in the first coordinate system of the three measurementpoints 521, 522, 523 (x_(iM1), y_(iM1), z_(iM1)) to determine anequation for the reference surface 550 (e.g., for a plane) having thefollowing form:

k _(0RS1) +k _(1RS1) ·x _(iRS1) +k _(2RS1) ·y _(iRS1) =z _(iRS1)  (2)

where (x_(iRS1), y_(iRS1), z_(iRS1)) are coordinates of anythree-dimensional point in the first coordinate system on the definedreference surface 550 and k_(0RS1), k_(1RS1), and k_(2RS1) arecoefficients obtained by a curve fitting of the three-dimensionalcoordinates in the first coordinate system.

It should be noted that a plurality of measurement points (i.e., atleast as many points as the number of k coefficients) are used toperform the curve fitting. The curve fitting finds the k coefficientsthat give the best fit to the points used (e.g., least squaresapproach). The k coefficients then define the plane or other referencesurface 550 that approximates the three-dimensional points used.However, if more points are used in the curve fitting than the number ofk coefficients, when you insert the x and y coordinates of the pointsused into the plane equation (2), the z results will generally notexactly match the z coordinates of the points due to noise and anydeviation from a plane that may actually exist. Thus, the x_(iRS1) andy_(iRS1) can be any arbitrary values, and the resulting z_(iRS1) tellsyou the z of the defined plane at x_(iRS1), y_(iRS1). Accordingly,coordinates shown in these equations can be for arbitrary points exactlyon the defined surface, not necessarily the points used in the fittingto determine the k coefficients.

In another embodiment, there are only two measurement points selectedfor a particular measurement (e.g., length, profile), prohibiting theuse of curve fitting based only on the three-dimensional coordinates ofthose two measurement points since three points are needed to determinek_(0RS1), k_(1RS1), and k_(2RS1). In that case, the video inspectiondevice 100 can identify a plurality of pixels proximate each of thepixels of the image corresponding to a plurality of points on the objectsurface 510 proximate each of the measurement points, and determine thethree-dimensional coordinates of those points, enabling curve fitting todetermine a reference surface 550.

In one embodiment and as shown in FIG. 8, the video inspection device100 can determine the three-dimensional coordinates in the firstcoordinate system of a plurality of frame points 560 (x_(iF1), y_(iF1),z_(iF1)) forming a frame 562 (e.g., a rectangle) on the referencesurface 550 around the anomaly 504 and the measurement points 521, 522,523, 524, which can be used later to display the location of thereference surface 550.

Once the reference surface 550 is determined, in the exemplaryembodiment shown in FIG. 8, the video inspection device 100 can conducta measurement (e.g., depth) of the anomaly 504 by determining thedistance between the fourth measurement point 524 selected to be at thedeepest point of the anomaly 504 and the reference surface 550. Theaccuracy of this depth measurement is determined by the accuracy inselecting the plurality of measurement points 521, 522, 523, 524 on theobject surface 510 of the viewed object 502. In many instances asdiscussed previously, the contour of the anomaly 504 in the image 500 isdifficult to assess from the two-dimensional image and may be too smallor otherwise insufficient to reliably locate the plurality ofmeasurement points 521, 522, 523, 524. Accordingly, in many cases, anoperator will want further detail in the area of the anomaly 504 toevaluate the accuracy of the location of these measurement points 521,522, 523, 524. So while some video inspection devices 100 can provide apoint cloud view of the full image 500, that view may not provide therequired level of detail of the anomaly 504 as discussed previously. Inorder to provide a more meaningful view of the object surface 510 in thearea around the measurement points 521, 522, 523, 524 than offered by apoint cloud view of the three-dimensional data of the entire image 500,the inventive method creates a subset of the three-dimensional data inthe region of interest.

At step 650, the CPU 150 of the video inspection device 100 canestablish a second coordinate system different from the first coordinatesystem. In one embodiment, the second coordinate system can be based onthe reference surface 550 and the plurality of measurement points 521,522, 523, and 524. The video inspection device 100 can assign the originof the second coordinate system (x_(O2), y_(O2), z_(O2))=(0, 0, 0) to belocated proximate the average position 525 of the three-dimensionalcoordinates of points on the reference surface 550 corresponding to twoor more of the plurality of measurement points 521, 522, 523, 524 on theobject surface 510 (e.g., by projecting the measurement points 521, 522,523, and 524 onto the reference surface 550 and determining an averageposition 525 on the reference surface 550). In some cases, thethree-dimensional coordinates of the points on the reference surface 550corresponding to the measurement points 521, 522, 523 can be the same.However, in some circumstances, due to noise and/or small variations inthe object surface 510, the measurement points 521, 522, 523 do not fallexactly on the reference surface 550, and therefore have differentcoordinates.

When determining points on the reference surface 550 that correspond tomeasurement points 521, 522, 523, 524 on the object surface 510, it isconvenient to apply the concept of line directions, which convey therelative slopes of lines in the x, y, and z planes, and can be used toestablish perpendicular or parallel lines. For a given line passingthrough two three-dimensional coordinates (x1, y1, z1) and (x2,y2,z2),the line directions (dx, dy, dz) may be defined as:

dx=x2−x1  (3)

dy=y2−y1  (4)

dz=z2−z1  (5)

Given a point on a line (x1, y1, z1) and the line's directions (dx, dy,dz), the line can be defined by:

$\begin{matrix}{\frac{( {x - {x\; 1}} )}{dx} = {\frac{( {y - {y\; 1}} )}{dy} = \frac{( {z - {z\; 1}} )}{dz}}} & (6)\end{matrix}$

Thus, given any one of an x, y, or z coordinate, the remaining two canbe computed. Parallel lines have the same or linearly scaled linedirections. Two lines having directions (dx1, dy1, dz1) and (dx2, dy2,dz2) are perpendicular if:

dx1·dx2+dy1·dy2+dz1·dz2=0  (7)

The directions for all lines normal to a reference plane defined usingequation (2) are given by:

dx _(RSN) =−k _(1RS)  (8)

dy _(RSN) =−k _(2RS)  (9)

dz _(RSN)=1  (10)

Based on equations (6) and (8) through (10), a line that isperpendicular to the reference surface 550 and passing through a surfacepoint (x_(S), y_(S), z_(S)) can be defined as:

$\begin{matrix}{\frac{x - x_{S}}{- k_{1\; {RS}}} = {\frac{y - y_{S}}{- k_{2\; {RS}}} = {z - z_{S}}}} & (11)\end{matrix}$

In one embodiment, the coordinates of a point on the reference surface550 (x_(iRS1), y_(iRS1), z_(iRS1)) corresponding to a point on theobject surface 510 (x_(iS1), y_(iS1), z_(iS1)) (e.g. three-dimensionalcoordinates in a first coordinate system of points on the referencesurface 550 corresponding to the measurement points 521, 522, 523, 524),can be determined by defining a line normal to the reference surface 550having directions given in equations (8)-(10) and passing through(x_(iS1), y_(iS1), z_(iS1)), and determining the coordinates of theintersection of that line with the reference surface 550. Thus, fromequations (2) and (11):

$\begin{matrix}{z_{iRS} = \frac{\begin{pmatrix}{{k_{1\; {RS}}^{2} \cdot z_{i\; S\; 1}} + {k_{1{RS}} \cdot x_{i\; S\; 1}} + {k_{2\; {RS}}^{2} \cdot z_{{iS}\; 1}} +} \\{{k_{2\; {RS}} \cdot y_{{iS}\; 1}} + k_{ORS}}\end{pmatrix}}{( {1 + k_{1\; {RS}}^{2} + k_{2\; {RS}}^{2}} )}} & (12) \\{x_{{iRS}\; 1} = {{k_{1\; {RS}\; 1} \cdot ( {z_{{iS}\; 1} - z_{{iRS}\; 1}} )} + x_{{iS}\; 1}}} & (13) \\{y_{{iRS}\; 1} = {{k_{2\; {RS}} \cdot ( {z_{{iS}\; 1} - z_{{iRS}\; 1}} )} + y_{{iS}\; 1}}} & (14)\end{matrix}$

In one embodiment, these steps (equations (3) through (14)) can be usedto determine the three-dimensional coordinates of points on thereference surface 550 corresponding to the measurement points 521, 522,523, 524. Then the average position 525 of these projected points of themeasurement points on the reference surface 550 (x_(M1avg), y_(M1avg),z_(M1avg)) can be determined. The origin of the second coordinate system(x_(O2), y_(O2), z_(O2))=(0, 0, 0) can then be assigned and locatedproximate the average position 525 (x_(M1avg), y_(M1avg), z_(M1avg)).

Locating the origin of the second coordinate system proximate theaverage position 525 in the area of the anomaly 504 with the z valuesbeing the perpendicular distance from each surface point to thereference surface 550 allows a point cloud view rotation to be about thecenter of the area of the anomaly 504 and permits any depth map colorscale to indicate the height or depth of a surface point from thereference surface 550.

In order to take advantage of this second coordinate system, at step660, the CPU 150 of the video inspection device 100 transforms thethree-dimensional coordinates in the first coordinate system (x_(i1),y_(i1), z_(i1)) determined for various points (e.g., the plurality ofsurface points, the plurality of measurement points 521, 522, 523, 524,the points on the reference surface 550 including the frame points 560,etc.) to three-dimensional coordinates in the second coordinate system(x_(i2), y_(i2), z_(i2)).

In one embodiment, a coordinate transformation matrix ([T]) can be usedto transform the coordinates according to the following:

([x _(i1) y _(i1) z _(i1)]−[x _(M1avg) y _(M1avg) z _(M1avg)])*[T]=[x_(i2) y _(i2) z _(i2)]  (15)

where [T] is a transformation matrix.

In non-matrix form, the three-dimensional coordinates in the secondcoordinate system can be determined by the following:

x _(i2)=(x _(i1) −x _(M1avg))*T ₀₀+(y _(i1) −y _(M1avg))*T ₁₀+(z _(i1)−z _(M1avg))*T ₂₀  (16)

y _(i2)=(x _(i1) −x _(M1avg))*T ₀₁+(y _(i1) −y _(M1avg))*T ₁₁+(z _(i1)−z _(M1avg))*T ₂₁  (17)

z _(i2)=(x _(i1) −x _(M1avg))*T ₀₂+(y _(i1) −y _(M1avg))*T ₁₂+(z _(i1)−z _(M1avg))*T ₂₂  (18)

where the transformation matrix values are the line direction values ofthe new x, y, and z axes in the first coordinate system.

At step 670, the CPU 150 of the video inspection device 100 determines asubset of the plurality of surface points that are within a region ofinterest on the object surface 510 of the viewed object 502. In oneembodiment, the region of interest can be a limited area on the objectsurface 510 of the viewed object 502 surrounding the plurality ofselected measurement points 521, 522, 523, 524 to minimize the amount ofthree-dimensional data to be used in a point cloud view. It will beunderstood that the step of determining of the subset 660 can take placebefore or after the transformation step 660. For example, if thedetermination of the subset at step 670 takes place after thetransformation step 660, the video inspection device 100 may transformthe coordinates for all surface points, including points that areoutside the region of interest, before determining which of those pointsare in the region of interest. Alternatively, if the determination ofthe subset at step 670 takes place before the transformation step 660,the video inspection device 100 may only need to transform thecoordinates for those surface points that are within the region ofinterest.

In one embodiment, the region of interest can be defined by determiningthe maximum distance (d_(MAX)) between each of the points on thereference surface 550 corresponding to the measurement points 521, 522,523, 524 and the average position 525 of those points on the referencesurface 550 (the origin of the second coordinate system (x_(O2), y_(O2),z_(O2))=(0, 0, 0) if done after the transformation, or (x_(M1avg),y_(M1avg), z_(M1avg)) in the first coordinate system if done before thetransformation). In one embodiment, the region of interest can includeall surface points that have corresponding points on the referencesurface 550 (i.e., when projected onto the reference surface) that arewithin a certain threshold distance (d_(ROI)) of the average position525 of the measurement points 521, 522, 523, 524 on the referencesurface 550 (e.g., less than the maximum distance (d_(ROI)=d_(MAX)) orless than a distance slightly greater (e.g. twenty percent greater) thanthe maximum distance (d_(ROI)=1.2*d_(MAX))). For example, if the averageposition 525 in the second coordinate system is at (x_(O2), y_(O2),z_(O2))=(0, 0, 0), the distance (d) from that position to a point on thereference surface 550 corresponding to a surface point (x_(iRS2),y_(iRS2), z_(iRS2)) is given by:

d _(iRS2)=√{square root over ((x _(iRS2) −x _(O2))²+(y _(iRS2) −y_(O2))²)}  (19)

Similarly, if the average position 525 in the first coordinate system isat (x_(M1avg), y_(M1avg), z_(M1avg)), the distance (d) from thatposition to a point on the reference surface 550 corresponding to asurface point (x_(iRS1), y_(iRS1), z_(iRS1)) is given by:

d _(iRS1)=√{square root over ((x _(iRS1) −x _(M1avg))²+(y _(iRS1) −y_(M1avg))²)}  (20)

If a surface point has a distance value (d_(iRS1) or d_(iRS2)) less thanthe region of interest threshold distance (d_(ROI)) and therefore in theregion of interest, the video inspection device 100 can write thethree-dimensional coordinates of that surface point and the pixel colorcorresponding to the depth of that surface point to a point cloud viewfile. In this exemplary embodiment, the region of interest is in theform of a cylinder that includes surface points falling within theradius of the cylinder. It will be understood that other shapes andmethods for determining the region of interest can be used.

The region of interest can also be defined based upon the depth of theanomaly 504 on the object surface 510 of the viewed object 502determined by the video inspection device 100 in the first coordinatesystem. For example, if the depth of the anomaly 504 was measured to be0.005 inches (0.127 mm), the region of interest can be defined toinclude only those points having distances from the reference surface550 (or z dimensions) within a certain range (±0.015 inches (0.381 mm))based on the distance of one or more of the measurement points 521, 522,523, 524 to the reference surface 550. If a surface point has a depthvalue inside the region of interest, the video inspection device 100 canwrite the three-dimensional coordinates of that surface point and thepixel color corresponding to the depth of that surface point to a pointcloud view file. If a surface point has a depth value outside of theregion of interest, the video inspection device 100 may not include thatsurface point in a point cloud view file.

At step 680, and as shown in FIG. 10, the monitor 170, 172 of the videoinspection device 100 can display a rendered three-dimensional view(e.g., a point cloud view) 700 of the subset of the plurality of surfacepoints in the three-dimensional coordinates of the second coordinatesystem, having an origin 725 at the center of the view. In oneembodiment (not shown), the display of the point cloud view 700 caninclude a color map to indicate the distance between each of the surfacepoints and the reference surface 750 in the second coordinate system(e.g., a first point at a certain depth is shown in a shade of redcorresponding that depth, a second point at a different depth is shownin a shade of green corresponding to that depth). The displayed pointcloud view 700 can also include the location of the plurality ofmeasurement points 721, 722, 723, 724. To assist the operator in viewingthe point cloud view 700, the video inspection device 100 can alsodetermine three-dimensional line points 771, 772, 773 along straightlines between two or more of the plurality of measurement points 721,722, 723 in the three-dimensional coordinates of the second coordinatesystem, and display those line points 771, 772, 773 in the point cloudview 700. The point cloud view 700 can also include a depth line 774from the measurement point 724 intended to be located at the deepestpoint of the anomaly 504 to the reference surface 750. In oneembodiment, the video inspection device 100 can determine if the depthline 774 exceeds a tolerance specification or other threshold andprovide a visual or audible indication or alarm of such an occurrence.

The displayed point cloud view 700 can also include a plurality of framepoints 760 forming a frame 762 on the reference surface 750 in thesecond coordinate system to indicate the location of the referencesurface 750. In another embodiment, the displayed point cloud view 700can also include a scale indicating the perpendicular distance from thereference surface 750.

As shown in FIG. 10, by limiting the data in the point cloud view 700 tothose points in the region of interest and allowing the view to berotated about a point 725 in the center of the region of interest (e.g.,at the origin), the operator can more easily analyze the anomaly 504 anddetermine if the depth measurement and placement of the measurementpoints 721, 722, 723, 724 was accurate. In one embodiment, the operatorcan alter the location of one or more of the measurement points 721,722, 723, 724 in the point cloud view 700 if correction is required.Alternatively, if correction is required, the operator can return to thetwo-dimensional image 500 of FIG. 8 and reselect one or more of themeasurement points 521, 522, 523, 524, and repeat the process.

In another embodiment, the monitor 170, 172 of the video inspectiondevice 100 can display a rendered three-dimensional view 700 of thesubset of the plurality of surface points in the three-dimensionalcoordinates of the first coordinate system without ever conducting atransformation of coordinates. In this embodiment, the point cloud view700 based on the original coordinates can also include the variousfeatures described above to assist the operator, including displaying acolor map, the location of the plurality of measurement points,three-dimensional line points, depth lines, frames, or scales.

FIG. 11 is a flow diagram of an exemplary method 800 for displaying atwo-dimensional image of viewed object simultaneously with an imagedepicting the three-dimensional geometry of the viewed object in anotherexemplary embodiment. It will be understood that the steps described inthe flow diagram of FIG. 11 can be performed in a different order thanshown in the flow diagram and that not all of the steps are required forcertain embodiments.

At step 810 of the exemplary method (FIG. 8), and as shown in FIG. 12,the video inspection device 100 (e.g., the imager 124 of FIG. 1) obtainsat least one two-dimensional image 903 of the object surface 911 of aviewed object 910 having an anomaly 912 and displays it on a first side901 of the display 900 (e.g., an integral display 170, external monitor172, or touch screen of a user interface). In one embodiment, thetwo-dimensional image 903 is displayed in a measurement mode of thevideo inspection device 100.

At step 820 of the exemplary method 800 (FIG. 11), and as shown in FIG.12, the video inspection device 100 (e.g., the CPU 150 of FIG. 1)determines the three-dimensional coordinates (e.g., (x, y, z)) of aplurality of surface points 913, 914 on the object surface 911 of theviewed object 910. In one embodiment, the video inspection devicegenerates three-dimensional data from the two-dimensional image 903 inorder to determine the three-dimensional coordinates. FIG. 12 is adisplay 900 of a two-dimensional first stereo image 903 of the viewedobject 910 on the first side 901 of the display 900, and a correspondingtwo-dimensional second stereo image 904 of the viewed object 910 on thesecond side 902 of the display 900. In one embodiment, the videoinspection device 100 (e.g., the CPU 150) employs stereo techniques todetermine the three-dimensional coordinates (e.g., (x, y, z)) of aplurality of surface points 913, 914 on the two-dimensional first stereoimage 903 by finding matching surface points 915, 916 on thecorresponding two-dimensional second stereo image 904 and then computingthe three-dimensional coordinates based on the pixel distance disparitybetween the plurality of surface points 913, 914 on the two-dimensionalfirst stereo image 903 (or an area of pixels (e.g., 4×4 area)) and thematching surface points 915, 916 on the corresponding two-dimensionalsecond stereo image 904. It will be understood and as shown in FIGS.12-14, the reference herein to a two-dimensional image with respect tostereo image 903, 904 can include both or either of the first (left)stereo image 903 and the second (right) stereo image 904.

Several different existing techniques can be used to provide thethree-dimensional coordinates of the surface points 913, 914 in thetwo-dimensional image 903 (FIG. 12) of the object surface 911 (e.g.,stereo, scanning systems, stereo triangulation, structured light methodssuch as phase shift analysis, phase shift moiré, laser dot projection,etc.). Most such techniques comprise the use of calibration data, which,among other things, includes optical characteristic data that is used toreduce errors in the three-dimensional coordinates that would otherwisebe induced by optical distortions. With some techniques, thethree-dimensional coordinates may be determined using one or moretwo-dimensional images captured in close time proximity that may includeprojected patterns and the like. It is to be understood that referencesto three-dimensional coordinates determined using two-dimensional image903 may also comprise three-dimensional coordinates determined using oneor a plurality of two-dimensional images of the object surface 911captured in close time proximity, and that the two-dimensional image 903displayed to the operator during the described operations may or may notactually be used in the determination of the three-dimensionalcoordinates.

At step 830 of the exemplary method 800 (FIG. 11), and as shown in FIGS.13 and 14, at least a portion of the two-dimensional image 903 of theviewed object 910 with measurement cursors 931, 932 is displayed on afirst side 901 of the display 900 and a rendered image 905 of thethree-dimensional geometry of at least a portion of the object surface911 of the viewed object 910 is displayed on the second side 902 of thedisplay 900. As compared to FIG. 12, the rendered image 905 replaces thesecond (right) stereo image 904 in the display 900. In one embodiment,the video inspection device 100 (e.g., the CPU 150) begins (and, in oneembodiment, completes) the process of determining the three-dimensionalcoordinates (e.g., (x, y, z)) of the plurality of surface points 913,914 on the object surface 911 of the viewed object 910 before theplacement and display of the measurement cursors 931, 932. Although theexemplary embodiments shown in FIGS. 13 and 14 show a single renderedimage 905 of the three-dimensional geometry of the object surface 911 ofthe viewed object 910 displayed on the second side 902 of the display900, it will be understood that more than one rendered image 905 can beshown simultaneously with or without the two-dimensional image 903.

In an exemplary embodiment shown in FIG. 13, the rendered image 905 is adepth profile image 906 showing the three-dimensional geometry of theobject surface 911 of the viewed object 910, including the anomaly 912.In another exemplary embodiment shown in FIG. 14, the rendered image 905is a point cloud view 907 showing the three-dimensional geometry of theobject surface 911 of the viewed object 910, including the anomaly 912.In the exemplary point cloud view 907 shown in FIG. 14, only a subset ofthe three-dimensional coordinates of the surface points 913, 914 on theobject surface 911 of the viewed object 910 are displayed in a region ofinterest based on the location of the measurement cursors 931, 932. Inanother embodiment, the point cloud view 907 displays all of thecomputed three-dimensional coordinates of the surface points 913, 914 onthe object surface 911 of the viewed object 910. In one embodiment,e.g., when the display is a user-interface touch screen, the user canrotate the point cloud view 907 using the touch screen.

In one embodiment and as shown in FIG. 14, the point cloud view 907 maybe colorized to indicate the distance between the surface points of theobject surface 911 of the viewed object 910 and a reference surface 960(e.g., reference plane determined using three-dimensional coordinatesproximate to one or more of the plurality of measurement cursors 931,932). For example, a first point at a certain depth is shown in a shadeof red corresponding that depth, a second point at a different depth isshown in a shade of green corresponding to that depth. A color depthscale 908 is provided to show the relationship between the colors shownon the point cloud view 907 and their respective distances from thereference surface 960. In one embodiment, the point could view 907 maybe surfaced to graphically smooth the transition between adjacent pointsin the point cloud view 907.

Once the three-dimensional coordinates have been determined for aplurality of surface points 913, 914 on the object surface 911 of theviewed object 910, the user can conduct measurements on thetwo-dimensional image 903.

In one embodiment, the video inspection device 100 saves as an image thesplit view of the two-dimensional image 903 and the rendered image 905.The video inspection device 100 can also save as metadata the original,full stereo image of the first (left) stereo image 903 and the second(right) stereo image 904 (e.g., grayscale only) as shown in FIG. 11 andthe calibration data to allow re-computation of the three-dimensionaldata and re-measurement from the saved file. Alternatively, the videoinspection device 100 can save the computed three-dimensionalcoordinates and/or disparity data as metadata, which reduces theprocessing time upon recall but results in a larger file size.

At step 840 of the exemplary method 800 (FIG. 11), and as shown in FIGS.13 and 14, measurement cursors 931, 932 are placed (using a pointingdevice) and displayed on the two-dimensional image 903 to allow thevideo inspection device 100 (e.g., the CPU 150) to determine thedimensions of the anomaly 912 (e.g., height or depth, length, width,area, volume, point to line, profile slice, etc.). In another embodimentwhere the two-dimensional image is not a stereo image, measurementcursors 931, 932 (as shown in FIGS. 13 and 14) can also be placed on thetwo-dimensional image 903 to allow the video inspection device 100(e.g., the CPU 150) to determine the dimensions of the anomaly 912(e.g., height or depth, length, width, area, volume, point to line,profile slice, etc.). In yet another embodiment, instead of being placedon the two-dimensional image 903, measurement cursors can be placed(using a pointing device) on the rendered image 905 of thethree-dimensional geometry of at least a portion of the object surface911 of the viewed object 910 on the second side 902 of the display 900.

In the exemplary display 900, the first measurement cursor 931 is placedon the first measurement point 921 on the object surface 911 of theviewed object 910 and the second measurement cursor 932 is placed on thesecond measurement point 922 on the object surface 911 of the viewedobject 910. Since the three-dimensional coordinates of the measurementpoints 921, 922 on the object surface 911 of the viewed object 910 areknown, a geometric measurement (e.g., depth or length measurement) ofthe object surface 911 can be performed by the user and the videoinspection device 100 (e.g., the CPU 150) can determine the measurementdimension 950 as shown in FIGS. 13 and 14. In the example shown in FIGS.13 and 14, a measurement line 933 is displayed on the two-dimensionalimage 903.

The rendered image 905 of the three-dimensional geometry of the objectsurface 911 of the viewed object 910 is displayed on the second side 902of the display 900 in order to assist in the placement of themeasurement cursors 931, 932 on the two-dimensional image 903 to conductthe geometric measurement. In a conventional system involving stereo ornon-stereo two-dimensional images, these measurement cursors 931, 932(as shown in FIGS. 13 and 14) are placed based solely on the viewprovided by the two-dimensional image 903, which may not allow foraccurate placement of the measurement cursors 931, 932 and accuratemeasurements.

At step 850 of the exemplary method 800 (FIG. 11), and as shown in FIGS.13 and 14, measurement identifiers 941, 942 corresponding to themeasurement cursors 931, 932 placed on the two-dimensional image 903 aredisplayed on the rendered image 905 of the three-dimensional geometry ofthe object surface 911 of the viewed object 912. For example, the firstmeasurement identifier 941 is shown on the rendered image 905 at thesame three-dimensional coordinate of the object surface 911 of theviewed object 912 as the first measurement cursor 931, and the secondmeasurement identifier 942 is shown on the rendered image 905 at thesame three-dimensional coordinate of the object surface 911 of theviewed object 912 as the second measurement cursor 932. In the exemplarypoint cloud view 907 shown in FIG. 14, a measurement line identifier 943corresponding to the measurement line 933 (e.g., depth measurement line)in the two-dimensional image 901 is displayed. This rendered image 905of the three-dimensional geometry of the object surface 911 of theviewed object 910 simultaneously displayed with the two-dimensionalimage 903 of the object surface 911 of the viewed object 912 allows theuser to more accurately place the measurement cursors 931, 932 toprovide a more accurate geometric measurement. In yet anotherembodiment, where the measurement cursors are placed (using a pointingdevice) on the rendered image 905, measurement identifiers correspondingto the measurement cursors are displayed on the two-dimensional image903.

In one embodiment, as the user changes the location of the measurementcursors 931, 932 in the two-dimensional image 903, the video inspectiondevice 100 (e.g., the CPU 150) automatically updates the location of themeasurement identifiers 941, 942 corresponding to the measurementcursors 931, 932 and the rendered image 905 (e.g., region of interest ordepth colors of the point cloud view 907 in FIG. 14) of thethree-dimensional geometry of the object surface 911 of the viewedobject 912 also changes to allow the user to visualize the newmeasurement virtually in real time. In another embodiment, after themeasurement cursors 931, 932 are placed in the two-dimensional image903, the measurement identifiers 941, 942 can be repositioned in therendered image 905.

In yet another embodiment, where the measurement cursors are placed(using a pointing device) on the rendered image 905 and measurementidentifiers corresponding to the measurement cursors are displayed onthe two-dimensional image 903, as the user changes the location of themeasurement cursors in the rendered image 905, the video inspectiondevice 100 (e.g., the CPU 150) automatically updates the location of themeasurement identifiers corresponding to the measurement cursors and thetwo-dimensional image also changes to allow the user to visualize thenew measurement virtually in real time. In another embodiment, after themeasurement cursors are placed on the rendered image 905. themeasurement identifiers can be repositioned in the two-dimensional image903.

At step 860 of the exemplary method 800 (FIG. 11), and as shown in FIGS.13 and 14, the video inspection device 100 (e.g., the CPU 150)determines the measurement dimension 950 sought by the user for theparticular geometric measurement (e.g., depth or length measurement)based on the locations of the measurement cursors 931, 932 and displaysthat measurement dimension 950 on the display 900. In anotherembodiment, the measurement dimension can displayed on the display 900on the rendered image 905.

As shown in FIGS. 12-14, soft keys 909 can be provided on the display900 to provide various functions to the user in obtaining images andtaking measurements (e.g., views, undo, add measurement, nextmeasurement, options, delete, annotation, take image, reset, zoom, fullimage/measurement image, depth map on/off, etc.). In one embodiment,when a user activates either the two-dimensional image 903 or therendered image 905, the particular soft keys 909 displayed can changebased on the active image.

In view of the foregoing, embodiments of the invention automaticallydetermine the depth or height of a point on an anomaly on a surface. Atechnical effect is to reduce the time required to perform themeasurement and to improve the accuracy of the measurement.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method, or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.), or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “service,” “circuit,” “circuitry,”“module,” and/or “system.” Furthermore, aspects of the present inventionmay take the form of a computer program product embodied in one or morecomputer readable medium(s) having computer readable program codeembodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

Program code and/or executable instructions embodied on a computerreadable medium may be transmitted using any appropriate medium,including but not limited to wireless, wireline, optical fiber cable,RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer (device), partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider).

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

This written description uses examples to disclose the invention,including the best mode, and also to enable any person skilled in theart to practice the invention, including making and using any devices orsystems and performing any incorporated methods. The patentable scope ofthe invention is defined by the claims, and may include other examplesthat occur to those skilled in the art. Such other examples are intendedto be within the scope of the claims if they have structural elementsthat do not differ from the literal language of the claims, or if theyinclude equivalent structural elements with insubstantial differencesfrom the literal language of the claims.

What is claimed is:
 1. A method for inspecting an object surface of aviewed object, the method comprising the steps of: displaying atwo-dimensional stereo image of the object surface on a display;determining the three-dimensional coordinates of a plurality of pointson the object surface using a central processor unit using stereotechniques; determining a rendered image of the three-dimensionalgeometry of at least a portion of the object surface using the centralprocessor unit; and simultaneously displaying the two-dimensional stereoimage and the rendered image on the display.
 2. The method of claim 1,wherein the two-dimensional image stereo is one of a first or a secondstereo image.
 3. The method of claim 1, further comprising the steps of:placing a plurality of measurement cursors on the two-dimensional stereoimage using a pointing device and displaying the plurality ofmeasurement cursors on the two-dimensional stereo image on the display;displaying on the rendered image a plurality of measurement identifierscorresponding to the measurement cursors on the two-dimensional stereoimage; and determining a measurement dimension of the object surfacebased on the locations of the plurality of measurement cursors on thetwo-dimensional stereo image using the central processor unit.
 4. Themethod of claim 1, further comprising the steps of: placing a pluralityof measurement cursors on the rendered image using a pointing device anddisplaying the plurality of measurement cursors on the rendered image onthe display; displaying on the two-dimensional image a plurality ofmeasurement identifiers corresponding to the measurement cursors on therendered image; and determining a measurement dimension of the objectsurface based on the locations of the plurality of measurement cursorson the rendered image using the central processor unit.
 5. The method ofclaim 1, wherein the rendered image is a depth profile image.
 6. Themethod of claim 1, wherein the rendered image is a point cloud view. 7.The method of claim 1, wherein the step of determining thethree-dimensional coordinates of a plurality of points on the objectsurface using a central processor unit begins before the step of placinga plurality of measurement cursors on the two-dimensional stereo image.8. The method of claim 7, wherein the step of determining thethree-dimensional coordinates of a plurality of points on the objectsurface using a central processor unit is completed before the step ofplacing a plurality of measurement cursors on the two-dimensional stereoimage.
 9. A device for inspecting an object surface of a viewed object,the device comprising: an elongated probe comprising an insertion tube;an imager located at a distal end of the insertion tube for obtaining atwo-dimensional stereo image of the object surface; a central processorunit for determining the three-dimensional coordinates of a plurality ofpoints on the object surface, and determining a rendered image of thethree-dimensional geometry of at least a portion of the object surface;and a display for simultaneously displaying the two-dimensional stereoimage and the rendered image.